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ABSTRACT 

This study explores the thought processes college students use when answering survey questions 
on standardized course/faculty evaluations. Thought processes are categorized as: System One 
or System Two, based on the framework developed by Kahneman (2003) and Stanovich and West 
(2000). System One processes are typically hurried, superficial, effortless, and charged with 
emotions. System Two processes are considered to be slower, more deliberate, thoughtful 
evaluations. Results from eight focus group discussions are provided. Student responses were 
categorized according to their adherence to System One or System Two cognitive processing. In 
total, 59% of the student responses were categorized as System One. Implications for faculty and 
higher education administrators are presented. 
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INTRODUCTION 



he process of having students evaluate their faculty was first begun in the 1920s, with the first 
published research on the topic by researchers at Purdue University (Remmers and Brandenburg 
1927). By the 1960s, the debate had begun on the plausibility, validity, and use of these student 
evaluations (Clayson 2009; Dommeyer, Baum, and Hanna 2002). While student ratings were first used to help 
administrators monitor teaching quality and to help teachers improve their teaching (Guthrie 1954), today they serve 
additional purposes that have more serious consequences. Many universities use student evaluations to serve as a 
basis for hiring new faculty, in promotion and tenure decisions, and in school accreditation reviews (Kulik 2001). 
Perhaps this is the fuel for the long-term debate over the use and reliance on student evaluations as the primary 
means by which to assess the performance of faculty. There are many studies that present both sides of various 
issues on whether or not student evaluations should still be used, however the need for faculty assessment remains 
(Clayson 2009; Kulik 2001). In fact, most of the debate about the appropriateness of having students evaluate 
faculty has not been about the need for this type of information, but rather about the process itself and the validity of 
the survey instruments used for this purpose (Clayson 2009; Merritt 2008). 


Thought Processes College Students Use When Evaluating Faculty 

While the majority of research about student’s evaluation of teaching (SET) deals with the issue of validity, 
(Clayson 2009), more recently studies have addressed the thought processes college student use when evaluating 
courses and teachers (Merritt 2008). By thought processes we are referring to the cognitive foundation of judgment, 
labeled in the psychology literature; System One and System Two (Kahneman 2003; Stanovich and West 2000). 
System One processes are typically hurried, superficial, effortless, and charged with emotions. System Two 
processes are considered to be slower, more deliberate, thoughtful evaluations. Although we might assume most 
adults are rational thinkers, weighing the pros and cons of issues and carefully evaluating them, in fact - most of our 
decisions result from System One thought processes. Kahneman (2003) asserts that most judgments involve strictly 
System One thought processes which are typically “fast and automatic,” “governed by habit,” and “difficult to 
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control or modify” (p. 698). According to Merritt (2008), adults generate intuitive judgments using System One’s 
emotional processes and then justify them through the cognitive processes of System Two. 

The dominance of System One thinking is seen in all aspects of daily life where we provide our “gut 
reaction to just about everything” from impulse buys at the checkout counter to the first impressions of people we 
meet (Merritt 2008; p. 276). These perspectives are reinforced by our “sound-bite culture” where we expect 
immediate information, on the spot answers, and instant gratification (Khalen 2009). 

These sound-bite judgments also occur on college campuses when students are asked to evaluate faculty 
and coursework. According to Merritt (2008), student evaluations of teaching begin, like most judgments, as 
System One processes, relying on first impressions or “thin-slice judgments” of the faculty member (Merritt 2008; 
p. 277). These “thin slices” are sometimes called brief observations of an individual that generate judgments about 
that individual’s personality, intentions, or other characteristics (Ambady, Bernieri, & Richeson 2000). From the 
moment a faculty member enters the classroom, students form a first impression and react to the instructor’s 
personality and non-verbal cues. In a study by Clayson and Sheffet (2006), instructors introduced themselves for 
less than five minutes on the first day of class, and then left the room turning the class over to a researcher. Students 
were then asked by the researcher to evaluate the faculty member on the Five-Factor Model of personality (Digman 
1990). Students’ perceptions of the instructor’s personality and subsequent evaluation of instruction were found to 
be strongly related. It appears that “students are universally associating perceived personality with instructional 
effectiveness” (Clayson and Sheffet 2006; p.156). In another study by Ambady and Rosenthal (1993), student 
evaluations of faculty “based on thirty-second exposures (to soundless videotapes of classroom instruction) were no 
more significant than judgments based on six-second clips” (p. 150). The authors concluded that students seem to 
make the same judgment regardless of the time exposure to the faculty member, suggesting that System One thought 
processes were in effect. 

Other research on faculty evaluations points to the risk of biases. Male instructors receive better evaluations 
than females (McPherson, Jewell, and Kim 2009), younger instructors are more popular than older ones 
(McPherson, Jewell, and Kim 2009), and physically attractive faculty are rated more positively (Campbell, Gerdes, 
and Steiner 2005). Evaluation scores have also been correlated with higher grades (Isley and Singh 2005; 
McPherson 2006; Pollio and Beck 2000). On a lighter note, even the distribution of chocolate prior to the evaluation 
process seems to result in higher scores (Youmans and Jee 2007). 

Given the overall findings of previous research, it appears that many students are dominated by System 
One processes when evaluating faculty effectiveness, rather than the more rational, cognitive approach. However, 
the majority of the research has measured the outcome or final evaluation of the instructor through quantitative 
means, i.e., self-report answers on surveys, without gathering data on the thought processes themselves. 

The current study expands on previous literature by exploring the type of thought processes college 
students use when answering questions on a student evaluation of teaching (SET) survey. Data is gathering using 
the qualitative approach of focus groups. Focus groups allow the researcher to go deeper into the emotional 
framework in which evaluations or judgments are being made (McDaniel and Gates 2004). 

METHODOLOGY 

In the current study eight focus groups from one public, northeastern university, were conducted with a 
total of 62 students in junior (45%) and senior (55%) classes, of which 51% were male. Of these students, forty 
were marketing majors, nineteen were management majors, and three were international business majors. The focus 
groups were conducted by research assistants, who acted as facilitators, and were approximately the same age as the 
group participants. In each session one facilitator acted as a note taker. The focus groups were conducted over 
period of one week and were totally voluntary. At the beginning of each session, a facilitator read the following 
instructions to the participants; 

In participating in today’s focus group discussion, we ask that you consider in general the college courses you have 
taken, not particular courses or instructors. We also ask that you do not disclose any professors ’ names or course 
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titles as this session is being recorded. If you speak about the contents of the focus group outside of this group, it is 
expected that you will not reveal to other people what individual participants said, in order to maintain your 
confidentiality. 

The sessions were also recorded on audio tape and took place in a social behavior lab with one-way 
mirrors. Both the written notes and the audio tapes were analyzed by the authors. Each session used a set of 
guiding questions, some of which correspond to the specific characteristics of the SET survey used at the institution. 
Due to the length of the focus group session of one hour, not all questions were addressed in each group. The list of 
these guiding questions appears in Table 1. 


Table 1: Teaching Evaluation Focus Group Guiding Questions 


1 . 

Do you think that it is important for students to evaluate professors? 

2. 

Do you take these surveys seriously? 

3. 

Does the time when evaluations are given affect their accuracy? 

4. 

Does the grade you are going to receive affect your evaluation of the course/teacher? 

5. 

SET Item: Rate the Overall Quality of the Course 

6. 

SET Item: Rate the Overall Quality of the Instructor 

7. 

SET Item: Rate the Logical Arrangement of the Course Material 


The guiding questions produced a plethora of responses during the eight hours of recorded sessions. The 
authors concluded that it would not be feasible to analyze each recorded response. Therefore, a sample of responses 
was drawn that was most representative of the student comments. Six independent judges categorized the sample 
responses according to System One or System Two thought processes. The training of these judges consisted of 
three steps: (1) Presenting an explanation of the focus groups and how personal statements were gathered and 
recorded; (2) Providing definitions and selected examples of Type One and Type Two human thought processes; (3) 
Explaining the coding process. Judges were given a list of 7 questions and 53 student responses to this set of 
questions. The judges were asked to read the student responses to each question and then indicate whether they 
perceived the response to be System One or System Two response. All 53 responses were evaluated by the six 
judges for the type of human thought process being used. Finally, inter-rater reliability was assessed using the 
methodology suggested by Trochim (2000). The inter-rater reliability was .92, which is above the threshold of .85 
suggested by Kassarjian (1977). Those responses whose categorization lacked agreement among the judges (4 out 
of 53 responses) were omitted from the analysis for a total of 49 responses. 

RESULTS 

To study the thought processes used by college students when answering questions on a student evaluation 
of teaching (SET) survey, a qualitative approach was used relying on focus group data. Student responses were 
categorized according to their adherence to System One or System Two cognitive processing. In total, 29 responses 
out of 49, or 59% of the responses, were categorized as System One. Differences were noted when looking at each 
question individually. For example, when students were asked “Rate the overall quality of the instructor,” 78% 
(seven out of nine) of the responses were judged to be System One. A similar result occurred with the question: 
“Rate the overall quality of the course” 71.4 percent (five out of seven) of the responses were evaluated as System 
One. Both of these results suggest that students are relying on emotionally-charged or gut reactions when forming 
their evaluations (see Table 2). For the question: “Rate the logical arrangement of the course material,” students 
tended to rely on System Two processing with 66.6% or four out of six responses reflecting more thoughtful 
deliberation. This result may be do, at least in part, to the type of question posed which encourages students to think 
logically about the structure and content of the course material. Some of the questions, such as “Do you take the 
SET surveys seriously?” produced an equal split between System One and System Two thought processes. In fact, 
anecdotal evidence in the focus group transcripts suggested that other factors such as class standing and gender may 
impact how serious students take the evaluation process. Some of the comments that lead the researchers to this 
conclusion were: “I didn’t care about SET’s when I first started - like as a freshman or even a sophomore, but now 
I really try to think about them when 1 do ‘em.” “Girls don’t take the SET’s seriously. They are too afraid to really 
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say what they think. In one class I had, the teacher was just awful, but the girl who sat next to me gave her all 
decent evals. I asked her why - she said she did not want to get the teacher in trouble.” For a complete listing of the 
guiding questions, sample responses, and Type One or Type Two designations see Table 2. 


Table 2: Categorization of Student Responses into System One and System Two Thought Processes 


SET Question/Issue 

Sample Student Responses 

System 

Do you think that it is 
important for students to 
evaluate professors? 

“They give us a grade. This is our evaluation back. 

1 


“It is our money. We are customers. We pay. We want the best instructor 
possible.” 

1 


“I am paying for it... If I learn nothing from the class; it will not help me on my 
job.” 

2 


“For the past 15 weeks, the teacher graded me. Now I get to grade them and 
show them how they have done.” 

1 


“Helps professor see if they are getting their point across.” 

2 


“Shows the professors the way they think they are doing something may not be 
how it is being portrayed or taken in.” 

2 


“How else would professors know how they are doing without feedback?” 

1 


“SET’s give students’ power.” 

1 




Do you take these surveys 
seriously? 

“Students do not feel that the results are of any use to them.” 

1 


“Many take them fast just to get done.” 

1 


“Too many SETs are given in a row - all in one week. By the last couple SETs 
students do not care what they rate.” 

2 


“If a big shot from the school of business came in to explain what they are 
for.. .people would take them more seriously.” 

2 


I take them seriously. If the professor did good or bad I want that to be 
expressed.” 

2 


"Not really. I feel that teachers can get by with terrible ratings and keep 
teaching. I know the ones that I gave crappy evals to and they are still here. So, 
what does that tell you?” 

1 




Does the grade you are going 
to receive affect the 
evaluation? 

“If a student feels it is their own fault a bad grade will have no effect - but many 
blame it on teacher.” 

2 


“The grade can be your fault or the professors. If students feel like it is the 
professor, it affects the eval.” 

2 


“It brings up the thought, ‘hey, I am doing bad’ and provokes revenge.” 

1 


“If you are getting a worse grade, you are often influenced to give lower ratings 
on the rest of the form.” 

1 


“Your grade predicts how you will answer.” 

1 


“Students who expect to get a bad grade will definitely rate the teacher lower.” 

1 


“Ownership of effort is a big thing.. .1 have blamed the teacher if I did not do as 
well as I hoped.” 

2 




Rate the logical arrangement 
of the course material 

“Makes me think did the teacher follow the syllabus or not.” 

2 


“I do not like when teachers change the syllabus.. .it affects my schedule.” 

1 


“Students rate lower when the professor does not follow the syllabus.” 

2 


“The class must follow some sort of order if it is not by chapter.” 

2 


“Skipping chapters is less organized.” 

1 


“Does the course flow and connect easily?” 

2 


58 




































American Journal of Business Education - March 2010 


Volume 3, Number 3 


Table 2: continued 


SET Question/Issue 

Sample Student Responses 

System 

Does the time when evals are 
given affect their accuracy? 

“Especially if it is right after you got a bad grade. Or if it is near the end of class 
or at a bad time of day for you.” 

1 


“Given late in the semester they seem ineffective because students do not see the 
changes.” 

2 


“Those given near finals are less accurate because the week is stressful.” 

2 


“Oh yea. Some people do rush to get out early... Especially if they give you 15 
or 20 minutes you just think about how soon you can get out” 

1 


“If at the beginning, you take your time and get things right.” 

2 


“I hate when professors do it last minute.” 

1 




Rate the overall quality of 
this course. 

"Do I like the material? That matters to me. I think the class is better - better 
quality - if I like the material.” 

1 


“The quality of the course depends on how the instructor teaches it.” 

2 


“If the class is boring I do not consider that a very good quality course.” 

1 


“If I’m not going to get my money’s worth, I think the course is a waste.” 

1 


“If it is a good course, I recommend it to my friends.” 

1 


“Do the students walk away with full knowledge of the subject matter?” 

2 


“Does the course relate to real-life?” 

1 




Rate the overall quality of 
the instructor. 

“Do I like them? If I really like a professor as a person - but they are a lousy 
teacher then I feel bad giving them a low grade... .1 just can’t give ‘em a low 
grade.” 

1 


“Easy professors get an A from me. 

1 


“If I like a professor, then I give them a 5, 6, or 7 no matter what.” 

1 


“I rate professors based on how they teach.” 

2 


“Teachers think their class is our only class; that’s sucks.” 

1 


“Do they know what they are talking about?” 

1 


“Does their teaching match the book material?” 

2 


“I take out my frustrations about teachers on the SET’s.” 

1 


“A Ph.D. means nothing to me.” 

1 


DISCUSSION AND IMPLICATIONS 

Student evaluations of teaching (SET) are not new to academia and the debate over the validity of such 
measures to assess the performance of faculty continues. At a minimum, colleges and universities expect students to 
take SET’s seriously and provide reasoned, thoughtful data that can be used by administration for numerous 
personnel issues, such as hiring, firing, and promotion and by faculty as feedback for continuous improvement. In 
reality, the results of our current study suggest that, at least in some cases, students are providing responses to SET’s 
that are completely the opposite. Often their evaluations are based on hurried thought processes charged with 
emotions. Consider the following illustrations from our study: “Teachers think their class is our only class; that’s 
sucks;” “I take out my frustration about teachers on the SET’s;” “Do I like the material? That matters to me. 1 
think the class is better - better quality - if 1 like the material; and “A Ph.D. means nothing to me.” 

As a result, “student evaluations of instruction appear to follow a seriously flawed paradigm” (Clayson and 
Sheffett 2006). Relying exclusively on these SET’s may lead to erroneous conclusions regarding course quality and 
instructor performance and ultimately, be detrimental to the faculty member’s career. Is good teaching simply what 
the students say it is? If so, then many good “instructors will not receive the high marks they deserve from their 
students for careful course planning, conscientious review of student work, and brilliantly conceived lectures 
(Machina 1987, p. 22). 

SET responses should be closely monitored both by faculty and administrators when they are used as 
indicators of teaching quality. Future research should investigate other forms of evaluation that could replace or be 
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used to supplement the current SET system. Given the potential impact on faculty careers and costs associated with 
hiring new faculty, it appears that more study is warranted on this topic. 
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